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-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

• Extensions of time may be available under the provisions of 37 CFR 1 .136(a), In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)S Responsive to communication(s) filed on 15 April 2005 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) I3 Claim(s) 20 - 39 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) K Claim(s) 20-39 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) Q Claim(s) are subject to, restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (0. 
a)D All b)D Some * c)Q None of: 

Certified copies of the priority documents have been received. 
Certified copies of the priority documents have been received in Application No. . 



2. D 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 

1 . Claims 20 - 39 are pending in this Office Action. 

Response to Arguments 

2. Applicants argue: "The MediaNet publication by Benitez et al. is unavailable as a 
reference" (page 7, paragraph 3). 

Regarding applicants' argument: Examiner respectfully disagrees with the 
applicants. However, Examiner wishes state that Applicants' rebuttal to Examiner's 
response to Rule 131 Declaration on August 19, 2004 states that the applicants 
disagree with the Office Action and reserve the right to submit further evidence to clarify 
this issues. However examiner still has not received any evidence to this effect. 
Therefore, Benitez et al. reference is still considered valid. 

It is important to recognize just what constitutes sufficient evidence to establish common 
ownership at the time the invention was made. The common ownership must be shown to 
exist at the time the later invention was made. A statement of present common ownership 
is not sufficient In re Onda, 229 USPQ 235 (Comm'r Pat. 1985). 

3. In view of the above, the examiner contends that all limitations as recited in the 
claims have been addressed in this Office Action. For the above reasons, Examiner 
believed that rejection of the last Office action was proper. 
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Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S:C. 103(a). 

5. Claims 20, 27 and 34 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Non-Patent Document: "Supporting Ranked Boolean Similarity Queries in MARS", 
IEEE Trans, on Knowledge and date Engineering, 10, Nov - Dec. 1998, Author: Ortega 
et at (hereinafter "Ortega") in view of Non-Patent Document: "MediaNet: A multimedia 
Information Network for Knowledge Representation", In Conference on Internet 
Multimedia Management Systems. Vol. 4210 (numbered Pages 1 - 12), Boston, MA, 
Nov. 2000, 1 ST/SPIE.00, Author: Benitez et al (hereinafter "Benitez"). 
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Regarding claims 20, 27 and 34, Ortega teaches a method for processing 
multimedia data in a computer system, comprising: 

receiving as input a high-level concept describing data to be accessed (see section 
1.2, page 4, paragraph 1, "A Boolean retrieval model (adapted for retrieval over images) 

is used to interpret the query being able to support such conceptual queries is 

critical for the versatility of large image databases.", and section 2, "In this section, we 
briefly describe the image features used . . . Other image features are available , 
however we restrict ourselves to queries involving only to the above features in this 
paper."); 

translating the high-level concept into a low-level query by using stored concept 
constructs which are defined using features derived from a plurality of application 
domains (see section 1.2, page 4, paragraph 1, "A Boolean retrieval model (adapted for 

retrieval over images) is used to interpret the query being able to support such 

conceptual queries is critical for the versatility of large image databases.", and section 2, 
"In this section, we briefly describe the image features used . . . Other image features 
are available , however we restrict ourselves to queries involving only to the above 
features in this paper."). 

Ortega does not explicitly teach transferring the low-level query to one or more 
search engines and concept repository. 

Benitez teaches transferring the low-level query to one or more search engines to 
access information using the low-level query (see section 3, page 6, "Typical content- 
based retrieval systems indexed ... or the value for any low-level features in the image 
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database" and section 4.2, page 9, "At this point and the results integrated into a 

unique list as described for visual queries"); and 

a concept repository for storing and accessing the concept constructs (see Fig.4 and 
section 4.1, page 7, "For each application, the list of concepts and relationships in the 
MediaNet knowledge based should be representative of the content in the database and 
the goal of the application task."). 

It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Benitez's teaching of "transferring the low-level query to one or more search engines to 
access information using the low-level query" and "concept repository" would have 
allowed Ortega's system to integrate both conceptual and perceptual representations of 
knowledge to impact a broad range of applications that deal with multimedia content at 
semantic and perceptual levels. This improves the performance of multimedia retrieval 
applications by using query expansion, refinement and translation across multiple 
content modalities as suggested by Benitez (see Abstract). 

6. Claims 21 - 26, 28 - 33, and 35 - 39 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Ortega in view of Benitez and further in view of Non-Patent 
Document: "CAMEL: Concept Annotated image Libraries", In Storage and Retrieval for 
Image and Video Database, San Jose, CA, Jan. 2001. SPIE (numbered pages 1 - 12), 
Author: Natsev et al (hereinafter "Natsev"). 
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Regarding claims 21 , 28 and 35, Ortega teaches storing matching algorithms in a 
matching algorithm library module (see section 4.8, page 20 - section 4.8.3, page 25) 
and Benitez teaches storing the concept constructs in a concept library module (see 
Fig. 4 and section 4.1, page 7, "For each application, the list of concepts and 
relationships in the MediaNet knowledge based should be representative of the content 
in the database and the goal of the application task"). 

Ortega or Benitez does not explicitly teach storing the features in a feature library 
module; and storing constraints in a constraint library module. 

However, Natsev further teaches storing the concept constructs in a concept library 
module (see section 1.2, "The concept cataloguing, or learning, phase is used to define 
visual concepts and build a concept library. Concepts are defined .... The concept 
library is module for a persistent storage of concepts"); 

storing the features in a feature library module (see fig. 3); and 

storing constraints in a constraint library module (see section 6, "Another 
improvement that we are considering is the introduction of spatial constraints in the 
query engine. For example, . . . , the corresponding document is ranked higher"). 

It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Natsev's teaching of storing the concept constructs in a concept library module, storing 
the features in a feature library module and storing constraints in a constraint library 
module would have presented anew direction for improving current image query 
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system, namely focusing the query specification part as gateway to better performance 
and usability (see Natsev section 6) for the combination Ortega and Benitez's system. 

Regarding claims 22, 29 and 36, Natsev teaches interfacing the library modules 
to the application domains (see fig. 3 and section 4, "Fig. 3 illustrates the architecture of 
the .... as well as the Concept Library module"). 

Regarding claims 23, 30 and 37, Natsev teaches building a concept construct 
(see fig. 1 and section 1.2, The concept cataloguing, or learning, phase is used to define 
visual concepts and build concept library. Concepts are defined . . . and to associate it 
with the given concept"). 

Regarding claims 24 and 31, Natsev teaches a method as defined in Claim 23, 
wherein the step of building a concept construct comprise combining one or more of the 
features with see section 1.2, The concept cataloguing, or learning, phase is used to 
define visual concepts and build concept library) 

zero or more of the stored concept (see Fig. 1 and section 1.2, "The concept 
library is a module for persistent storage of concept"), and 

zero or more of the constraints (see section 6, "Another improvement that we are 
considering is the introduction of spatial constraint in query engine"). 
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Regarding claims 25 and 32, Ortega teaches a concept construct is represented 
using a hierarchical fuzzy graph data tree-structure comprising: nodes that correspond 
to child-concepts and a subset of the features; aggregation edges that correspond to 
parent-child relationships; and association edges that correspond to inter-sibling 
constraints (see sections 4. 1 and 4.2, pages 1 0 and 11). 

Regarding claims 26 and 33, Natsev teaches a method as defined in Claim 20, 
wherein the features are user defined (see section 2, "One of the most commonly used 
image features is color histogram, . . . places the burden on the user to specify weights 
for the different features"). 

Regarding claim 38, Natsev teaches a system as defined in Claim 34, wherein 
the translation engine further comprises an interpreter that translates the high level 
concept (see fig.1). 

Regarding claim 39, Natsev teaches a system as defined in Claim 34, further 
comprises a search engine (see section 2, "the high-level approach that . . . . , there are 
some considerations that make certain query engines more suitable than others". 
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Conclusion 

7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Fred I. Ehichioya whose telephone number is 571-272- 
4034. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John E. Breene can be reached on 571-272-4107. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 



Application/Control Number: 09/925,397 



Page 10 



Art Unit: 2162 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Fred I. Ehichioya 
Patent Examiner 
Art Unit 2162 

July 5, 2005 s\ 
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The amendment document filed on 25 April 2005 is considered non-compliant because it has failed to meet the 
requirements of 37 CFR 1.121. In order for the amendment document to be compliant, correction of the following item(s) is 
required. 

THE FOLLOWING MARKED (X) ITEM(S) CAUSE THE AMENDMENT DOCUMENT TO BE NON-COMPLIANT: 

□ 1 . Amendments to the specification: 

□ A. Amended paragraph(s) do not include markings. 

□ B. New paragraph(s) should not be underlined. 

□ C. Other . 

□ 2. Abstract: 

□ A. Not presented on a separate sheet. 37 CFR 1.72. 

□ B. Other . 

□ 3. Amendments to the drawings: 

□ A. The drawings are not properly identified in the top margin as "Replacement Sheet," "New Sheet," or 

"Annotated Sheet" as required by 37 CFR 1.121(d). 

□ B. The practice of submitting proposed drawing correction has been eliminated. Replacement drawings 

showing amended figures, without markings, in compliance with 37 CFR 1.84 are required. 

□ C. Other . 

^ 4. Amendments to the claims: 

A. A complete listing of all of the claims is not present. 

□ B. The listing of claims does not include the text of all pending claims (including withdrawn claims) 

^ C. Each claim has not been provided with the proper status identifier, and as such, the individual status 
of each claim cannot be identified. Note: the status of every claim must be indicated after its claim 
number by using one of the following status identifiers: (Original), (Currently amended), (Canceled), 
(Previously presented), (New), (Not entered), (Withdrawn) and (Withdrawn-currently amended). 

□ D. The claims of this amendment paper have not been presented in ascending numerical order. 

□ E. Other: . 

For further explanation of the amendment format required by 37 CFR 1.121, see MPEP § 714 and the USPTO website at 
http://www.uspto.gov/web/offices/pac/dapp/opla/preoqnotice/officeflver.pdf . 

TIME PERIODS FOR FILING A REPLY TO THIS NOTICE: 

1. Applicant is given no new time period if the non-compliant amendment is an after-final amendment or an amendment 
filed after allowance. If applicant wishes to resubmit the non-compliant after-final amendment with corrections, the 
entire corrected amendment must be resubmitted within the time period set forth in the final Office action. 

2. Applicant is given one month, or thirty (30) days, whichever is longer, from the mail date of this notice to supply the 
corrected section of the non-compliant amendment in compliance with 37 CFR 1.121, if the non-compliant 
amendment is one of the following: a preliminary amendment, a non-final amendment (including a submission for a 
request for continued examination (RCE) under 37 CFR 1.114), a supplemental amendment filed within a suspension 
period under 37 CFR 1.103(a) or (c), and an amendment filed in response to a Quayle action. 

Extensions of time are available under 37 CFR 1.136(a) only if the non-compliant amendment is a non-final 
amendment or an amendment filed in response to a Quayle action. 

Failure to timely respond to this notice will result in: 

Abandonment of the application if the non-compliant amendment is a non-final amendment or an amendment 
filed in response to a Quayle action; or 

Non-entry of the amendment if the non-compliant amendment is a preliminary amendment or supplemental 
amendment. 



U.S. Patent and Trademark Office 
PTOL-324 (11-04) 



Notice of Non-Compliant Amendment (37 CFR 1.121) 



Part of Paper No. 07052005 



